Lidar-based high definition map generation

ABSTRACT

Various arrangements for generating a high definition map are presented. An image of a roadway environment may be captured. A laser imaging detection and ranging point cloud of the roadway environment may be generated. An object recognition process may be performed on the image of the roadway environment to detect one or more objects present in the roadway environment. A fusion process may be performed using the lidar point cloud to identify a location of the one or more detected objects from the image in a vehicle frame of reference. The one or more objects may be mapped from the vehicle frame of reference to a global frame of reference An autonomous driving high definition map may be created that includes the mapped one or more objects in the global frame of reference.

BACKGROUND

For autonomous driving systems and driver-assistance systems to safelypilot vehicles, it may be beneficial for the systems to have access to ahigh accuracy map of the roadways which the vehicle is navigating. Thehigh accuracy map may provide the autonomous driving system with anindication of what objects are present nearby, such as ahead on aroadway and hidden around a turn. Such a high accuracy map may includesignificantly more information than a typical navigational map. Forinstance, a high accuracy map may indicate the presence of trafficlights at an intersection, where roadway signs, lane markings, curbs,traffic islands, etc., are located. Generation of such a high accuracymap may typically be labor-intensive. For example, a human may manuallyreview and tag objects present in images captured of a roadwayenvironment. While such a manual arrangement may be acceptable for smallgeographic regions such as for testing purposes, such an arrangement maynot be practical for large-scale generation of high accuracy maps, suchas nationwide or worldwide.

SUMMARY

Various embodiments are described related to a method for generating anautonomous driving high definition map. In some embodiments, a methodfor generating an autonomous driving high definition map is described.The method may include capturing, using a camera system installed on avehicle, an image of a roadway environment. The method may includecreating, using a laser imaging detection and ranging (lidar) systeminstalled on the vehicle, a lidar point cloud of the roadwayenvironment. The lidar point cloud may be mapped to the image of theroadway environment. The method may include performing an objectrecognition process on the image of the roadway environment to detectone or more objects present in the roadway environment. The method mayinclude performing a fusion process using the lidar point cloud toidentify a location of the one or more detected objects from the imagein a vehicle frame of reference. The method may include mapping the oneor more objects from the vehicle frame of reference to a global frame ofreference using global navigation satellite system (GNSS) data. Themethod may include creating the autonomous driving high definition mapthat comprises the mapped one or more objects in the global frame ofreference.

Embodiments of such a method may include one or more of the followingfeatures: Performing the object recognition process may includeperforming a trained deep learning process on the image of the roadwayenvironment to detect the one or more objects present in the roadwayenvironment. Performing the object recognition process may includeperforming the trained deep learning process by an onboard processingsystem of the vehicle. Performing the object recognition process mayinclude performing the trained deep learning process by a remote highdefinition map server system. The method may include calibrating thelidar system with the camera system such that distance measurements madeusing the lidar system correspond to determined image positions withinimages captured by the camera system. Performing the object recognitionprocess on the image of the roadway environment to detect the one ormore objects may include identifying a permanent object. The permanentobject may be a permanent stationary object that may be expected toremain stationary over time. The method may include identifying animpermanent object. The impermanent object may be moveable and may beexpected to move over time. The method may include, in response toidentifying the impermanent object, removing the impermanent object frominclusion in the autonomous driving high definition map. The method mayinclude performing the object recognition process on the image of theroadway environment to detect a two-dimensional item present in theroadway environment that may be included in the autonomous driving highdefinition map. The method may include mapping the two-dimensional itemfrom the vehicle frame of reference to the global frame of referenceusing global navigation satellite system (GNSS) data. The method mayfurther include capturing, using a GNSS system installed on the vehicle,the GNSS data. The method may further include driving, using an onboardautonomous driving system, a second vehicle using the created autonomousdriving high definition map.

In some embodiments, a system for generating an autonomous driving highdefinition map is described. The system may include a global navigationsatellite system (GNSS) sensor. The system may include a camera systemthat captures an image of a roadway environment. The system may includea laser imaging detection and ranging (lidar) system that creates alidar point cloud of the roadway environment. The system may include aprocessing system, comprising one or more processors, that may beconfigured to perform object recognition on the image of the roadwayenvironment to detect one or more objects present in the roadwayenvironment. The system may be configured to perform a fusion processusing the lidar point cloud to identify a location of the one or moredetected objects from the image in a vehicle frame of reference. Theprocessing system may be configured to map the one or more objects fromthe vehicle frame of reference to a global frame of reference using datafrom a GNSS sensor. The system may be configured to create theautonomous driving high definition map that comprises the mapped one ormore objects in the global frame of reference.

Embodiments of such a system may include one or more of the followingfeatures: The processing system being configured to perform the objectrecognition process may include the processing system being configuredto perform a trained deep learning process on the image of the roadwayenvironment to detect the one or more objects present in the roadwayenvironment. The GNSS sensor, the camera system, the lidar system, andthe processing system may be installed on-board a vehicle. The GNSSsensor, the camera system, and the lidar system may be installedon-board a vehicle and the processing system may be part of the remoteserver system. The processing system may be further configured tocalibrate the lidar system with the camera system such that distancemeasurements made using the lidar system correspond to determined imagepositions within images captured by the camera system. The processingsystem may be further configured to identify a permanent object. Thepermanent object may be a permanent stationary object that may beexpected to remain stationary over time. The processing system may befurther configured to identify an impermanent object. The impermanentobject may be moveable and may be expected to move over time. Theprocessing system may be further configured to remove the impermanentobject from inclusion in the autonomous driving high definition map inresponse to identifying the impermanent object. The processing systemmay be further configured to perform the object recognition process onthe image of the roadway environment to detect a two-dimensional itempresent in the roadway environment that may be included in theautonomous driving high definition map. The processing system may befurther configured to map the two-dimensional item from the vehicleframe of reference to the global frame of reference using data from theGNSS sensor.

In some embodiments, an apparatus for generating a high definition mapof a roadway is described. The apparatus may include a means forcapturing an image of a roadway environment. The apparatus may include ameans for creating a distance point cloud of the roadway environment.The apparatus may include a means for performing an object recognitionprocess on the image of the roadway environment to detect one or moreobjects present in the roadway environment. The apparatus may include ameans for performing a fusion process using the distance point cloud toidentify a location of the one or more detected objects from the imagein a vehicle frame of reference. The apparatus may include a means formapping the one or more objects from the vehicle frame of reference to aglobal frame of reference. The apparatus may include a means forcreating the autonomous driving high definition map that comprises themapped one or more objects in the global frame of reference.

Embodiments of such an apparatus may include one or more of thefollowing features: The apparatus may further include means forperforming the object recognition process on the image of the roadwayenvironment to detect a two-dimensional item present in the roadwayenvironment that may be included in the autonomous driving highdefinition map. The apparatus may include means for mapping thetwo-dimensional item from the vehicle frame of reference to the globalframe of reference.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of an embodiment of a vehicle-basedsystem for gathering and processing data about a roadway environment.

FIG. 2 illustrates a block diagram of an embodiment of a system forbuilding a high definition map using data gathered by the vehicle-basedsystem.

FIG. 3 illustrates an image that may be captured and object recognitionprocessing that may be performed on the image.

FIG. 4 illustrates an embodiment of a method for generating highdefinition map data for operating an autonomous vehicle.

FIG. 5 illustrates an embodiment of a method for using high definitionmap data for operating an autonomous vehicle.

DETAILED DESCRIPTION

A high definition map may include information that an autonomous drivingsystem or driver-assistance system can use to operate or drive a vehiclewith an increased amount of safety and/or efficiency. For example, ahigh definition map may include: locations of lane boundaries;indications of street signs; fixed obstacles (e.g., curbs, trafficislands, guardrails, bridge supports); traffic lights; etc. To generatesuch a high definition map, a vehicle may be driven (either manually bya human driver or autonomously) on a roadway while an onboard vehiclesystem is used to capture information about the roadway environment. Thecaptured information may then be processed locally onboard the vehicleand/or remotely at a remote server system to generate a high definitionmap. The high definition map may then be used for controlling a vehiclethat is driving on the roadway.

To generate the high definition map, one or more cameras installed onthe vehicle may be used to capture images of the roadway environment.Additionally installed on the vehicle may be a lidar (light detectionand ranging) system that can measure the distance from the lidar systemto objects present in the roadway environment. The one or more camerasand the lidar system may be calibrated such that a point to which adistance is measured by the lidar system can be mapped to a locationwith an image captured by the camera. An image recognition process maybe performed on captured images to identify particular types of objects,such as pedestrians, vehicles, traffic lights, signs, obstacles, etc.Impermanent objects, such as vehicles and pedestrians, may be filteredout and not used for creating the high definition map. Using acombination of the lidar distance measurements and the recognizedobjects from the captured images, the location of particular types ofobjects may be determined in reference to the vehicle. Global navigationsatellite system data may be used to convert the location of the objectsfrom the vehicle's frame of reference to a global frame of reference.

The mapped identified objects may then be stored as part of a databaseof high definition map data and may be accessed or otherwise provided toan autonomous driving system on a vehicle. The high definition map maybe used for autonomous driving of a vehicle. An “autonomous drivingsystem” refers to a system that can drive, operate, or pilot a vehiclefor a period of time without human input being needed to control thevehicle. The high definition map data may also be used by a“driver-assistance system.” A driver-assistance system may perform atleast some of the tasks that are typically performed by a human driveror serve as a safety failsafe for situations in which a human driver hasperformed a likely mistaken or incorrect action while driving (e.g.,failing to brake for a red traffic light, drifting out of a lane,failing to stop or slow down by an appropriate distance from an obstaclein the path of the driver's vehicle).

Further detail regarding such embodiments is provided in relation to thefigures. FIG. 1 illustrates a block diagram of an embodiment 100 of avehicle-based system for gathering and processing data about a roadwayenvironment. Embodiment 100 may include: vehicle 101; onboard vehicleprocessing system 120; vehicle sensors 130; network interface 140;antenna 150; cellular network 160; network 170; and map server system180.

Vehicle 101 can refer to various forms of vehicles. Vehicle 101 may be apassenger car, pickup truck, sport utility vehicle, truck, motorizedcart, all-terrain vehicle, motorcycle, powered scooter, or some otherform of powered vehicle. Such vehicles may be configured to becontrolled by a human driver (hereinafter a “driver”), an autonomousdriving system (or driver-assistance system), or both. Therefore, atleast in some vehicles, a driver may control the vehicle, while at othertimes the autonomous driving system may control the vehicle.

Vehicle sensors 130 can include: camera 131, IMU (inertial measurementunit) 132, lidar module 133; and GNSS (global navigation satellitesystem) module 134. As part of vehicle sensors 130, camera 131 may bepresent. In some embodiments, more than one camera may be present.Multiple cameras may have different or overlapping fields-of-views. Insome embodiments, the angle of field-of-view is different, such as forshort-range and long-range cameras. Camera 131 may be a visible lightcamera that has a field-of-view of the environmental scene in front ofvehicle 101. Lidar module 133 may be used to determine the distance toobjects in the roadway environment of vehicle 101. Camera 131, lidarmodule 133, and onboard vehicle processing system 120 may be calibratedsuch that a lidar measurement can be mapped to a particular locationwithin an image captured by camera 131. Lidar module 133 may capture apoint cloud that represents distances from lidar module 133 to thenearest object in a variety of directions. Therefore, for a givencaptured image, multiple points (e.g., tens, hundreds) from a capturedpoint cloud may be mapped to different locations with the image. Thesepoints are representative of the measured distance from the vehicle orlidar module to objects present within the image.

GNSS module 134 may use one or more GNSS satellite systems to determinea precise location of GNSS module 134 and, thus, by extension, vehicle101 on which GNSS module 134 is installed. GNSS module 134 may use GPS,GLONASS, Galileo, BeiDou (BDS) or some other form of navigationsatellite system to determine a location of vehicle 101. IMU 132 may beused to determine the speed and direction of vehicle 101. This data maybe used alternatively or in addition to speed and direction dataobtained from GNSS module 134.

Onboard vehicle processing system 120 may receive data from vehiclesensors 130. Onboard vehicle processing system 120 may furthercommunicate with map server system 180 through network interface 140 andantenna 150. Onboard vehicle processing system 120 may include variouscomputerized components, such as one or more processors andcommunication busses. The one or more processors used as part of onboardvehicle processing system 120 may include one or more specific-purposeprocessors that have various functionality hardcoded as part of the oneor more processors, such as an application-specific integrated circuit(ASIC). Additionally or alternatively, one or more general-purposeprocessors may be used as part of onboard vehicle processing system 120that execute stored instructions that cause the general-purposeprocessors to perform specific-purpose functions. Therefore, softwareand/or firmware may be used to perform at least some of the functions ofonboard vehicle processing system 120. Further detail regarding thefunctioning of onboard vehicle processing system 120 is provided inrelation to FIG. 2.

In some embodiments, onboard vehicle processing system 120 performsprocessing on captured images from camera 131 and processing onpoint-cloud data received from lidar module 133. Onboard vehicleprocessing system 120 may be used to perform an object recognitionprocess on captured images to identify one or more types of objects.Onboard vehicle processing system 120 may map distances measured usingthe lidar module 133 to locations in captured images. The absolutelocation of objects may be determined by analyzing location dataobtained from GNSS module 134 to objects identified in the images anddistances measured using lidar module 133. In other embodiments, some orall of this processing may be performed remotely at map server system180.

Network interface 140 may be used to facilitate communication betweenonboard vehicle processing system 120 and various external sources. Insome embodiments, network interface 140 uses antenna 150 to wirelesslycommunicate with cellular network 160, which may be a 3G, 4G, 5G, orsome other form of wireless cellular network. Cellular network 160 mayuse one or more networks 170, which can include the Internet, tocommunicate with a remote map server system 180. Map server system 180may be operated by an entity that creates and stores high definition mapdata for use by autonomous vehicles. For instance, map server system 180may be operated by (or have operated on its behalf) a manufacturer orprovider of autonomous vehicles or autonomous driving services.Therefore, map server system 180 may communicate with a large number(e.g., thousands) of autonomous driving systems 110 deployed ingeographically-scattered vehicles. Network interface 140 may also beable to communicate with other forms of wireless networks. For instance,network interface 140 may be used to communicate with a wireless localarea network (WLAN), such as a Wi-Fi network to which on-board vehicleprocessing system 120 has permission to access. For example, when parkedat a home or office, vehicle 101 may be within range of a Wi-Fi network,through which the Internet and map server system 180 may be accessed.Other forms of network-based communication with map server system 180are possible, such as a Bluetooth communication link via a vehicleoccupant's mobile device to a cellular network or WLAN. In otherembodiments, rather than wirelessly transmitting data to map serversystem 180, data captured using vehicle sensors 130 may be storedlocally onboard vehicle 101, such as to a solid state drive or otherform of non-transitory processor-readable medium. The captured data maythen be transferred to the map server system, such as via a wiredcommunication arrangement or by a removable form of non-transitoryprocessor-readable medium being used (e.g., flash memory, solid statedrive).

FIG. 2 illustrates a block diagram of an embodiment of a system 200 forbuilding a high definition map using data gathered by the vehicle-basedsystem. System 200 represents various components that may be implementedusing specialized hardware or software executed by one or moregeneral-purpose processors, for example, one or more specific-purposeprocessors that have various functionalities hardcoded as part of theone or more processors, such as an ASIC. Further, the various componentsof system 200 may be part of onboard vehicle processing system 120 ormap server system 180. In some embodiments, the functionality of somecomponents may be part of onboard vehicle processing system 120 whileothers are performed remotely as part of map server system 180.

Camera image 205 may be received for processing. A camera image may bereceived periodically, such as every 500 ms. Each camera image may beinitially processed using object recognition engine 210. Objectrecognition engine 210 may be trained to recognize various types ofobjects. Such types of objects, such as indicated by exemplary objecttype list 215, can include: vehicles; pedestrians; traffic lights; fixedstructures; lane marking; curbs; fixed obstacles; traffic islands;traffic signs; etc. Object recognition engine 210 may use a neuralnetwork or other form of deep-learning-based object recognition module.If object recognition engine 210 is based on deep learning, objectrecognition engine 210 may have initially been provided with a large setof images that have the object types that are desired to be identifiedproperly tagged. This set of images may be used to train objectrecognition engine 210 to properly recognize instances of the same typesof objects. Once properly trained and tested, object recognition engine210 may operate on received images without human intervention ormonitoring. That is, object recognition engine 210 may be able torecognize the trained object types without a human manually tagging thetrained object types. In some embodiments, a human may perform somelevel of review to confirm that each object or some objects werecorrectly located and tagged.

Of the trained object types, both impermanent and permanent objects arepresent. Impermanent objects are objects that are either currentlymoving or are expected to move over time. For example, vehicles andpedestrians are types of impermanent objects. At any given time, avehicle or pedestrian may not be moving; however, it can be expectedthat the pedestrian and vehicle will at some point move and are notobstacles that should be included in a high definition map. Permanentobjects are objects that are not expected to move over time. Forexample, traffic lights, traffic signs, bridge supports, curbs, andwalls are types of permanent objects. Such objects can be expected to befixed in position unless roadway construction changes the configurationof permanent objects.

Impermanent object removal engine 245 may serve to tag or otherwiseselect impermanent objects that are to be removed from inclusion in thefinal generated high definition map data. Impermanent object removalengine 245 may be configured to remove all types of impermanent objects.These impermanent objects include vehicles and pedestrians. Impermanentobject removal engine 245 may be reconfigured to include additional orfewer types of impermanent objects.

Lidar and image fusion engine 225 may serve to fuse lidar data 230obtained from lidar module 133 and objects present in captured imagesthat have not been removed by impermanent object removal engine 245.Lidar data 230 may be in the form of a point cloud that includesdistance measurements in a direction in which the distance measurementwas made. Lidar data 230 may be captured at the same time or a similartime as the image with which the lidar data is being fused by lidar andimage fusion engine 225. That is, while camera 131 is capturing animage, lidar module 133 may be capturing a point cloud representative ofdistances to various locations present within the image. Therefore, inorder for the point cloud to be accurately representative of thedistances to various locations within the image, the point cloud may becaptured within a threshold time of when the image was captured, such as100 ms. Lidar and image fusion engine 225 may be calibrated such thatparticular points from within the captured point cloud are mapped tolocations within the captured image. By using these mapped locations,the distance to the objects identified by object recognition engine 210within the captured images can be determined. The output of lidar andimage fusion engine 225 may be indications of tagged objects along withrelative distances and directions from the vehicle (or the lidar module)to the tagged objects.

The output of lidar and image fusion engine 225 may be passed to globallocation engine 235. Global location engine 235 may receive GNSS data240 from GNSS module 134. Global location engine 235 may convert thedistance and direction data of identified objects from the vehicle frameof reference to a global frame of reference. The received GNSS data 240may indicate a precise location in the form of global coordinates. Theseglobal coordinates may be obtained at the same or approximately the sametime as lidar data 230 and camera image 205 were obtained. In someembodiments, the global coordinates may be obtained within a thresholdperiod of time, such as 100 ms, of when camera image 205 and lidar data230 were obtained. Using the global coordinates and the distance anddirection to the object from the vehicle, global location engine 235 maydetermine the global location of the identified objects. The locationand type of these objects may be output as high definition map data. Thelocation and type of these objects may be added to a high definition mapdatabase that may be later accessed to help control a vehicle performingautonomous driving.

In some situations, one or more types of objects, such as indicated inlist 220, may not be fused with lidar data. For instance, some types ofobjects may not be classified as an obstacle (referred to as“non-obstacle objects”). For example, lane boundaries on a roadway maybe a non-obstacle object that is essentially flat on the roadway and maynot function as an obstacle because a vehicle can drive over them.Object recognition engine 210 may identify lane boundaries and/or othernon-obstacle objects, and may pass information about the lane boundariesdirectly to global location engine 235. Based upon the location of thelane boundaries and/or other non-obstacle objects within camera image205 as detected by object recognition engine 210, global location engine235 may determine the location of the lane boundaries and/or othernon-obstacle objects within a global frame of reference. The locationand type of these non-obstacle objects may be output as high definitionmap data and stored to the high definition map database for later use incontrolling a vehicle performing an autonomous driving. The output ofsystem 200 can include object types such as in exemplary list 250.

FIG. 3 illustrates an image 300 that may be captured and objectrecognition processing that may be performed on the image. Image 300 mayrepresent an image that has been processed by object recognition engine210. Object recognition engine 210 may identify: traffic lights 310;vehicles 320; and lane markings 330. Data from a lidar point cloud andGNSS data may be used to determine the absolute location of some of theidentified objects in image 300. For instance, the absolute location oftraffic lights 310 may be determined. The location of vehicles 320 maybe discarded since vehicles 320 are impermanent objects. The location oflane markings 330 may be determined using image 300 and GNSS data(without lidar data).

Various methods may be performed using the systems and devices of FIGS.1 and 2. FIG. 4 illustrates an embodiment of a method 400 for generatinghigh definition map data for operating an autonomous vehicle. Each blockof method 400 may be performed using system 200. Components of system200 may be incorporated as part of onboard vehicle processing system 120or map server system 180. Therefore, performance of method 400 may beperformed at onboard vehicle processing system 120, map server system180, or a combination of onboard vehicle processing system 120 and mapserver system 180.

Further, it should be understood that no portion of method 400 ismanually performed by a person. Rather, all of method 400 is performedwithout a need for a human to provide input for the high definition mapdata to be created. In some embodiments, a human may drive a vehicle towhich the lidar module, GNSS module, and camera are installed to gatherthe data to create the high definition map. In other embodiments, thevehicle may be autonomously driven, possibly without a human onboard.

At block 405, the optical camera may be calibrated with the lidarmodule. This calibration process may be used to map particular points(measurements in particular directions) of the lidar point cloud withlocations in images captured by the optical camera. Once calibrated,distance measurements from the lidar point cloud may be used todetermine the distance to particular objects within images captured bythe optical camera.

At block 410, one or more images of the roadway environment may becaptured from a vehicle that is traveling on the roadway. The image maybe time stamped. Simultaneously or within a threshold period of timeearlier or later than the capturing of the one or more images, a lidarpoint cloud of the roadway environment may be created based on lidarmeasurements made from the vehicle at block 415. Each point within thepoint cloud may have a particular direction and distance. The lidarpoint cloud may also be associated with the timestamp. At the same timeor within a threshold period of time earlier or later than the capturingof the one or more images, at block 420, a GNSS module may be used todetermine an absolute position of a GNSS module present on the vehicle,and therefore can be used as indicative of the vehicle's absolutelocation. The GNSS data may also be associated with the timestamp. Thetimestamps of the lidar point cloud, the captured image, and the GNSSdata may be compared to determine whether all of such data was capturedwithin a threshold period of time.

At block 425, an object recognition process may be performed on thecaptured image. The object recognition process may be performed toidentify particular types of objects for which the object recognitionprocess has been trained. A deep learning process may be used to trainthe object recognition process. For example, a neural network may becreated that can identify various types of objects that are of interestfor high definition map creation. At block 430, object types that havebeen classified as impermanent objects, such as vehicles andpedestrians, may be removed from consideration for inclusion in theoutput high definition map data. Such impermanent objects and theirassociated locations may be ignored for the remainder of method 400.

At block 435, lidar point cloud data may be fused with the recognizedobjects in the captured images. Since the lidar module was previouslycalibrated with the optical camera, distance measurements from the lidarpoint cloud may be used to determine the distance from the vehicle toobjects identified at block 425 within the image captured at block 410.At block 435, the distance and direction from the vehicle to identifiedpermanent objects may be determined. At block 440, GNSS data may be usedto map objects from the vehicle frame of reference to a global frame ofreference. Therefore, following block 440, the absolute locations ofobjects identified using the object recognition process of block 425 maybe determined.

At block 445, GNSS data may be used to map other object types that werenot fused with lidar point cloud data at block 435. For example, lanemarkings may be a type of object identified at block 425 for which anabsolute location is identified using GNSS data in addition to thelocation of the lane markings within the captured image. Sense lanemarkings are essentially two-dimensional, lidar data may not be used andthe lane marking can be expected to be present on the roadway surface.

At block 450, the high definition map data may include the mappedobjects in the global frame of reference. At block 455, this highdefinition map data may be stored for later retrieval for use forautonomous driving activities. For example, when a vehicle approaches aregion, high definition map data for the region may be provided to thevehicle for use by an onboard autonomous driving system. Such highdefinition map data may allow the onboard autonomous driving system toanticipate where traffic lights, traffic signs, and obstacles areexpected to be present.

FIG. 5 illustrates an embodiment of a method 500 for using highdefinition map data for operating an autonomous vehicle. Method 500 maybe performed using high definition map data created using method 400.Method 500 may be performed using the systems and devices of FIG. 1. Atblock 505, an absolute position of a vehicle may be determined usingGNSS measurements made using a GNSS module installed on a vehicle. Thevehicle may be performing autonomous driving or some form of driverassistance. At block 510, a request to the map server system may be madethat requests high definition map data for a region relative to theabsolute position of the vehicle determined at block 505. The highdefinition map data requested at block 510 may have been created as partof method 400. In some embodiments, rather than the high definition mapdata being requested at block 510 from a map server system, the highdefinition map data may be stored locally on a non-transitory processorreadable medium by an onboard vehicle processing system present on avehicle.

High definition map data may be received at block 515 from the mapserver system in response to the request of block 510. At block 520, thevehicle may be driven autonomously at least partially based on thereceived high definition map data. This high definition map data mayhelp an onboard autonomous driving system detect the location ofpermanent objects that affect how and where the vehicle is driven. Forexample, the location of a traffic light can be anticipated based onhigh definition map data indicating that a traffic light is present atan upcoming intersection.

The methods, systems, and devices discussed above are examples. Variousconfigurations may omit, substitute, or add various procedures orcomponents as appropriate. For instance, in alternative configurations,the methods may be performed in an order different from that described,and/or various stages may be added, omitted, and/or combined. Also,features described with respect to certain configurations may becombined in various other configurations. Different aspects and elementsof the configurations may be combined in a similar manner. Also,technology evolves and, thus, many of the elements are examples and donot limit the scope of the disclosure or claims.

Specific details are given in the description to provide a thoroughunderstanding of example configurations (including implementations).However, configurations may be practiced without these specific details.For example, well-known circuits, processes, algorithms, structures, andtechniques have been shown without unnecessary detail in order to avoidobscuring the configurations. This description provides exampleconfigurations only, and does not limit the scope, applicability, orconfigurations of the claims. Rather, the preceding description of theconfigurations will provide those skilled in the art with an enablingdescription for implementing described techniques. Various changes maybe made in the function and arrangement of elements without departingfrom the spirit or scope of the disclosure.

Also, configurations may be described as a process which is depicted asa flow diagram or block diagram. Although each may describe theoperations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be rearranged. A process may have additional steps notincluded in the figure. Furthermore, examples of the methods may beimplemented by hardware, software, firmware, middleware, microcode,hardware description languages, or any combination thereof. Whenimplemented in software, firmware, middleware, or microcode, the programcode or code segments to perform the necessary tasks may be stored in anon-transitory computer-readable medium such as a storage medium.Processors may perform the described tasks.

Having described several example configurations, various modifications,alternative constructions, and equivalents may be used without departingfrom the spirit of the disclosure. For example, the above elements maybe components of a larger system, wherein other rules may takeprecedence over or otherwise modify the application of the invention.Also, a number of steps may be undertaken before, during, or after theabove elements are considered.

What is claimed is:
 1. A method for generating an autonomous drivinghigh definition map, the method comprising: capturing, using a camerasystem installed on a vehicle, an image of a roadway environment;creating, using a laser imaging detection and ranging (lidar) systeminstalled on the vehicle, a lidar point cloud of the roadwayenvironment, wherein the lidar point cloud is mapped to the image of theroadway environment; performing an object recognition process on theimage of the roadway environment to detect one or more objects presentin the roadway environment, wherein performing the object recognitionprocess comprises performing a trained deep learning process on theimage of the roadway environment to detect a plurality of objectspresent in the roadway environment; identifying one or more detectedobjects of the plurality of detected objects from the trained deeplearning process that are classified as impermanent; performing a fusionprocess using the lidar point cloud to identify a location of the one ormore detected objects from the image in a vehicle frame of reference,wherein the identified one or more detected objects classified asimpermanent are ignored from the fusion process; mapping the one or moreobjects from the vehicle frame of reference to a global frame ofreference using global navigation satellite system (GNSS) data; andcreating the autonomous driving high definition map that comprises themapped one or more objects in the global frame of reference.
 2. Themethod for generating the autonomous driving high definition map ofclaim 1, wherein performing the object recognition process comprisesperforming the trained deep learning process by an onboard processingsystem of the vehicle.
 3. The method for generating the autonomousdriving high definition map of claim 1, wherein performing the objectrecognition process comprises performing the trained deep learningprocess by a remote high definition map server system.
 4. The method forgenerating the autonomous driving high definition map of claim 1,further comprising: calibrating the lidar system with the camera systemsuch that distance measurements made using the lidar system correspondto determined image positions within images captured by the camerasystem.
 5. The method for generating the autonomous driving highdefinition map of claim 1, wherein performing the object recognitionprocess on the image of the roadway environment to detect the one ormore objects comprises: identifying a permanent object, wherein thepermanent object is a permanent stationary object that is expected toremain stationary over time.
 6. The method for generating the autonomousdriving high definition map of claim 5, further comprising: performingthe object recognition process on the image of the roadway environmentto detect a two-dimensional item present in the roadway environment thatis to be included in the autonomous driving high definition map; andmapping the two-dimensional item from the vehicle frame of reference tothe global frame of reference using global navigation satellite system(GNSS) data.
 7. The method for generating the autonomous driving highdefinition map of claim 1, further comprising capturing, using a GNSSsystem installed on the vehicle, GNSS data.
 8. The method for generatingthe autonomous driving high definition map of claim 1, furthercomprising driving, using an onboard autonomous driving system, a secondvehicle using the created autonomous driving high definition map.
 9. Asystem for generating an autonomous driving high definition map, thesystem comprising: a global navigation satellite system (GNSS) sensor; acamera system that captures an image of a roadway environment; a laserimaging detection and ranging (lidar) system that creates a lidar pointcloud of the roadway environment; and a processing system, comprisingone or more processors, that is configured to: perform objectrecognition on the image of the roadway environment to detect one ormore objects present in the roadway environment, wherein the processingsystem being configured to perform an object recognition processcomprises the processing system being configured to perform a traineddeep learning process on the image of the roadway environment to detecta plurality of objects present in the roadway environment; identify oneor more detected objects of the plurality of detected objects from thetrained deep learning process that are classified as impermanent;perform a fusion process using the lidar point cloud to identify alocation of the one or more detected objects from the image in a vehicleframe of reference, wherein the identified one or more detected objectsclassified as impermanent are ignored from the fusion process; map theone or more objects from the vehicle frame of reference to a globalframe of reference using data from the GNSS sensor; and create theautonomous driving high definition map that comprises the mapped one ormore objects in the global frame of reference.
 10. The system forgenerating the autonomous driving high definition map of claim 9,wherein the GNSS sensor, the camera system, the lidar system, and theprocessing system are installed on-board a vehicle.
 11. The system forgenerating the autonomous driving high definition map of claim 9,wherein the GNSS sensor, the camera system, and the lidar system areinstalled on-board a vehicle and the processing system is part of aremote server system.
 12. The system for generating the autonomousdriving high definition map of claim 9, wherein the processing system isfurther configured to calibrate the lidar system with the camera systemsuch that distance measurements made using the lidar system correspondto determined image positions within images captured by the camerasystem.
 13. The system for generating the autonomous driving highdefinition map of claim 9, wherein the processing system is furtherconfigured to: identify a permanent object, wherein the permanent objectis a permanent stationary object that is expected to remain stationaryover time.
 14. The system for generating the autonomous driving highdefinition map of claim 13, wherein the processing system is furtherconfigured to remove the impermanent object from inclusion in theautonomous driving high definition map in response to identifying theimpermanent object.
 15. The system for generating the autonomous drivinghigh definition map of claim 14, wherein the processing system isfurther configured to: perform the object recognition process on theimage of the roadway environment to detect a two-dimensional itempresent in the roadway environment that is to be included in theautonomous driving high definition map; and map the two-dimensional itemfrom the vehicle frame of reference to the global frame of referenceusing data from the GNSS sensor.
 16. An apparatus for generating a highdefinition map of a roadway, the apparatus comprising: a means forcapturing an image of a roadway environment; a means for creating adistance point cloud of the roadway environment; a means for performingan object recognition process on the image of the roadway environment todetect one or more objects present in the roadway environment, whereinthe means for performing the object recognition process comprises ameans for performing a trained deep learning process on the image of theroadway environment to detect a plurality of objects present in theroadway environment; a means for identifying one or more detectedobjects of the plurality of detected objects from the trained deeplearning process that are classified as impermanent; a means forperforming a fusion process using the distance point cloud to identify alocation of the one or more detected objects from the image in a vehicleframe of reference; a means for mapping the one or more objects from thevehicle frame of reference to a global frame of reference, wherein theidentified one or more detected objects classified as impermanent areignored from the fusion process; and a means for creating an autonomousdriving high definition map that comprises the mapped one or moreobjects in the global frame of reference.
 17. The apparatus forgenerating the high definition map of the roadway of claim 16, theapparatus further comprising: means for performing the objectrecognition process on the image of the roadway environment to detect atwo-dimensional item present in the roadway environment that is to beincluded in the autonomous driving high definition map; and means formapping the two-dimensional item from the vehicle frame of reference tothe global frame of reference.